Image processing systems and methods for displaying multiple images of a biological specimen

ABSTRACT

A system and method of displaying of multiple simultaneous views of a same region of a biological tissue sample. Logical instructions are executed by a processor to perform operations such as receiving a plurality of images of the biological tissue sample, converting the plurality of images to a common reference frame based on the individual metadata of each image, and arranging the plurality of images into a display pattern for simultaneous viewing of different aspects of the imaged biological tissue sample on a display screen. The plurality of images is produced by preprocessing images of the biological tissue sample. Each image shows a view mode of a same region of the biological tissue sample, and each image contains metadata that describe spatial orientation, such as the translation, rotation, and magnification, of the image to bring the plurality of images to a common view.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is a continuation of International PatentApplication No. PCT/EP2016/070105 filed Aug. 25, 2016, which claimspriority to and the benefit of U.S. Provisional Patent Application No.62/213,487, filed Sep. 2, 2015. Each of the above patent applications isincorporated herein by reference as if set forth in its entirety.

APPLICATIONS BACKGROUND OF THE SUBJECT DISCLOSURE Field of the SubjectDisclosure

The present subject disclosure relates to imaging for medical diagnosis.More particularly, the present subject disclosure relates to the displayand transformation of field of view (FOV) images in unison.

Background of the Subject Disclosure

In the analysis of biological specimens such as tissue sections, blood,cell cultures and the like, biological specimens are stained with one ormore combinations of stains, and the resulting assay is viewed or imagedfor further analysis. Observing the assay enables a variety ofprocesses, including diagnosis of disease, assessment of response totreatment, and development of new drugs to fight disease. An assayincludes one or more stains conjugated to an antibody that binds toprotein, protein fragments, or other objects of interest in thespecimen, hereinafter referred to as targets or target objects. Theantibodies or other compounds that bind a target in the specimen to astain are referred to as biomarkers in this subject disclosure. Somebiomarkers have a fixed relationship to a stain (e.g., the often usedcounterstain hematoxylin), whereas for other biomarkers, a choice ofstain may be used to develop and create a new assay. Subsequent tostaining, the assay may be imaged for further analysis of the contentsof the tissue specimen. An image of an entire slide is typicallyreferred to as a whole-slide image, or simply whole-slide.

Typically, in immunoscore computations, a scientist uses a multiplexassay that involves staining one piece of tissue or a simplex assay thatinvolves staining adjacent serial tissue sections to detect or quantify,for example, multiple proteins or nucleic acids etc. in the same tissueblock. With the stained slides available, the immunological data, forinstance, the type, density and location of the immune cells, can beestimated from the tumor tissue samples. It has been reported that thisdata can be used to predict the patient survival of colorectal cancerand demonstrates important prognostic role.

In the traditional workflow for immunoscore computation, the expertreader such as a pathologist or biologist selects the representativefields of view (FOVs) or regions of interest (ROIs) manually, as theinitial step, by reviewing the slide under a microscope or reading animage of a slide, which has been scanned/digitized, on a display. Whenthe tissue slide is scanned, the scanned image is viewed by independentreaders and the FOVs are manually marked based on the readers' personalpreferences. After selecting the FOVs, the computer produces counts ofimmune cells via an automatic algorithm in each FOV, or apathologist/reader manually counts the immune cells within the selectedFOVs. Manual selection of the FOVs and counting is highly subjective andbiased to the readers, as different readers may select different FOVs tocount. Hence, an immunoscore study is no longer reproducible. Byautomating the selection of the fields of view, a uniform method isapplied reducing the subjectivity of independent readers. Use oflow-resolution images to perform the FOV selection furthermore improvescomputational efficiency, allowing the analyst to rapidly proceed toanalysis of the tissue regions.

It is often the case that any single view of a tissue sample may lead toseveral possible diagnoses of disease state. A tedious examination ofseveral different views must rely on the memory of the expert reader inorder to narrow the focus on any particular diagnosis.

Prior art includes, for example, US2003/0210262 by Graham et al. thatgenerally teaches displaying at least two views of the same region on amicroscope slide adjacent to each other, where the views offer differingillumination conditions, and the viewing device offers similarrectilinear translations.

Lastly, US2012/0320094 by Ruddle et al. generally teaches displaying atleast two microscope slide images of the same region, adjacent to eachother on a viewing screen, at different magnifications.

The automatic identification of FOVs is disclosed in U.S. 62/005,222,and PCT/EP2015/062015 the entirety being incorporated by referenceherewith.

SUMMARY OF THE SUBJECT DISCLOSURE

The present invention provides an image processing method for displayingmultiple images of a biological tissue region and a respective imageprocessing system as claimed in the independent claims. Embodiments ofthe invention and further aspects of the invention are provided in thefurther dependent and independent claims.

A ‘tissue sample’ as understood herein is any biological sample obtainedfrom a tissue region, such as a surgical biopsy specimen that isobtained from a human or animal body for anatomic pathology. The tissuesample may be a prostrate tissue sample, a breast tissue sample, a colontissue sample or a tissue sample obtained from another organ or bodyregion.

A ‘multi-channel image’ as understood herein encompasses a digital imageobtained from a biological tissue sample in which different biologicalstructures, such as nuclei and tissue structures, are simultaneouslystained with specific fluorescent dyes, each of which fluoresces in adifferent spectral band thus constituting one of the channels of themulti-channel image. The biological tissue sample may be stained by aplurality of stains and/or by a stain and a counterstain, the laterbeing also refered to as a “single marker image”.

An ‘unmixed image’ as understood herein encompasses a grey-value orscalar image obtained for one channel of a multi-channel image. Byunmixing a multi-channel image one unmixed image per channel isobtained.

A ‘color channel’ as understood herein is a channel of an image sensor.For example, the image sensor may have three color channels, such as red(R), green (G) and blue (B).

A ‘heat map’ as understood herein is a graphical representation of datawhere the individual values contained in a matrix are represented ascolors.

‘Thresholding’ as understood herein encompasses the application of apredefined threshold or sorting of local maxima to provide a sorted listand selecting of a predetermined number of the local maxima from the topof the sorted list.

‘Spatial low pass filtering’ as understood herein encompasses a spatialfiltering using a spatial filter that performs a low pass filteringoperation on a neighborhood of image pixels, in particular a linear ornon-linear operation. In particular, spatial low pass filtering may beperformed by applying a convolutional filter. Spatial filtering is assuch known from the prior art, (cf. Digital Image Processing, ThirdEdition, Rafael C. Gonzalez, Richard E. Woods, page 145, chapter 3.4.1).

‘Local maximum filtering’ as understood herein encompasses a filteringoperation where a pixel is considered a local maximum if it is equal tothe maximum value in a subimage area. Local maximum filtering can beimplemented by applying a so called max filter, (cf. Digital ImageProcessing, Third Edition, Rafael C. Gonzalez, Richard E. Woods, page326, chapter 5).

A ‘field of view (FOV)’ as understood herein encompasses an imageportion that has a predetermined size and shape, such as a rectangularor circular shape.

In accordance with embodiments of the invention a tissue region of acancer biopsy tissue sample is sliced into neighboring tissue slices.The tissue slices may be marked by single or multiple stains for theidentification of respective biological features. A digital image isacquired from each of the marked tissue slices by means of an imagesensor that has a number of color channels, such as an RGB image sensor.

An image registration algorithm is performed with respect to theacquired multiple digital images. Various suitable image registrationalgorithms that are as such known from the prior art can be used forperforming the image registration, (cf.https://en.wikipedia.org/wiki/Image_registration andhttp://tango.andrew.cmpu.edu/˜gustavor/42431-intro-bioimaging/readings/ch8.pdf).In particular, an affine transformation can be utilized to perform theimage registration.

The image registration algorithm generates a geometrical transformationthat aligns corresponding points of the images. The geometricaltransformation can be provided in the form of mappings, where eachmapping maps the points of one of the images to corresponding points ofanother one of the images.

The images are aligned in accordance with the image registration. Inother words, the geometrical transformations that are generated by theimage registration algorithm are applied to the images for aligning theimages in order to display the aligned images on a display in atwo-dimensional plane. As a result the display shows the multiple imagesafter registration and alignment such that each one of the images thatare displayed in the two-dimensional plane shows a matching tissueregion.

An image transformation command can be entered via a graphical userinterface with respect to one of the displayed images, such as byperforming a mouse click on the image, rotating a mouse wheel orperforming a gesture that is entered via a touch-sensitive displayscreen. For example, the image transformation command is a command tozoom in or zoom out, to rotate or perform another image transformationsuch as by selecting a field of view.

In response to the entry of the image transformation command totransform the one of the displayed images the other images aresimultaneously transformed in the same way. This is done using thegeometrical transformations, such as the mappings, that have beengenerated by the image registration algorithm. As a consequence, theimage transformation is executed in unison in response to the imagetransformation command in all of the images.

Embodiments of the present invention are particularly advantageous as auser, such as a pathologist, can readily view and manipulate imagesobtained from tissue slices of a tissue region in an intuitive way thatfacilitates the task of performing a diagnosis.

In accordance with embodiments of the invention at least one of thetissue slices is marked by multiple stains for the acquisition of amulti-channel image. The multi-channel image is unmixed to provide a setof unmixed images. The unmixed images do not need to be registered withrespect to each other or with respect to the multi-channel image as theyare all based on the identical dataset that is acquired by the opticalsensor from one of the tissue slices. The multi-channel image isselected as a reference image for performing the image registrationalgorithm with respect to the multiple images, excluding the set ofunmixed images. This provides a mapping of each one of the multipleimages to the reference image, except for the unmixed images.

Using the multi-channel image as a reference image for the imageregistration is advantageous as it reduces the computational cost ofperforming the image registration and the alignment of the images as noimage registration and alignment is required for the unmixed images

In accordance with an embodiment of the invention the imagetransformation command is a zoom in or a zoom out command that isreceived via the graphical user interface using gesture recognition. Forexample, the user's gesture by which the zoom in or zoom out imagetransformation command is entered is a pinch gesture that is performedby placing two fingers onto one of the displayed images. The imagetransformation command is thus received with respect to the one of thedisplayed images on which the user places his or her fingers and isexecuted with respect to this image and also synchronously with respectto the other displayed images.

In accordance with a further embodiment of the invention the acquiredmultiple images are stored on a server computer. The images aretransmitted from the server computer to a mobile battery-poweredtelecommunication device, such as a smartphone or mobile computer, via atelecommunication network for displaying the images on a display of thetelecommunication device. This provides an utmost degree of flexibilityas regards access and viewing of the images.

In accordance with an embodiment of the invention at least the executionof the image registration algorithm is performed by the server computerand the resultant geometrical transformation, such as the mappings, aretransmitted together with the images from the server computer to thetelecommunication device. This may be advantageous as the imageregistration algorithm may require substantial computational processingpower. Executing the image registration algorithm as a preprocessingstep by the server computer and not on the mobile battery-poweredtelecommunication device has the advantage of saving battery power andreducing the latency time experienced by the user.

In accordance with embodiments of the invention one or more fields ofview are defined automatically in one or more of the images. A graphicalsymbol, such as a rectangular box, may be displayed in order to indicatethe location of the field of view in one of the images. A user may enteran image transformation command with respect to a field of view byselecting the respective graphical symbol such as by touching thegraphical symbol on a touch-sensitive display. In response to theselection of the graphical symbol a zoom in image transformation may beexecuted with respect to the field of view and sychnronously withrespect to aligned image portions in the other images.

The automatic definition of the fields of view may also be performed bythe server computer in order to reduce the computational burden of thetelecommunication device, thus increasing battery lifetime anddecreasing latency times. In this instance meta data that is descriptiveof the defined fields of view is generated by the server computer andtransmitted together with the images via the network in order to enablethe telecommunication device to display the graphical symbol indicatingthe location of a field of view defined by the server computer.

In accordance with a further aspect of the invention an image processingsystem is provided that is configured to execute a method of theinvention.

The present invention is surprisingly effective to allow a coordinatedreview of a multiplicity of diagnostic images of the same tissue regionthat are shown adjacent to one another on a single viewing screen. Allimages are aligned and scaled to a common reference frame, and they canall be translated and zoomed together, each showing an important aspectof histology. This enables a more directed and determined diagnosis ofimportant conditions, where any single image might only support a moretentative conclusion from an expert reader.

The present invention has at least the following advantageous featuresand robustness:

1. A common display reference frame is chosen and used for imagevisualization.

2. The preprocessed images of the biological tissue sample are convertedto the common display reference frame by constructing a destination viewfor each preprocessed image in order to produce displayable images.

3. User gestures are accepted to dynamically alter the common displayreference frame. For example, the images can be simultaneouslytranslated, rotated, or zoomed in magnification.

4. When each image shows a different staining to highlight importantaspects of the biological tissue sample, the simultaneous views offer amore certain diagnosis of tissue conditions than could be had by relyingon the memory of the expert reader conducting a serial inspection ofthese same images.

The present invention further accommodates images that are derived fromconsecutive microtome slices, where they may require rotation inaddition to translation to align common features of interest. Also, thepresent invention may involve tagging images with metadata to describetheir location in a tissue section, and this this information is usedfor construction of affine transforms to adjust the images to a commonreference frame for display. Additionally, the present invention allowsfor simultaneous zooming in magnification of all images at the samescale.

In one embodiment, the subject disclosure features a system ofsimultaneously displaying multiple views of a same region of abiological tissue sample. The system may comprise a processor and amemory coupled to the processor. The memory can store computer-readableinstructions that, when executed by the processor, cause the processorto perform operations.

In another embodiment, the subject disclosure features a method ofsimultaneously displaying multiple views of a same region of abiological tissue sample. The method may be implemented by an imaginganalysis system and may be stored on a computer-readable medium. Themethod may comprise logical instructions that are executed by aprocessor to perform operations.

In some embodiments, the operations may include receiving a plurality ofpreprocessed images of the biological tissue sample, choosing a commondisplay reference frame that is used for image visualization, convertingthe plurality of preprocessed images to the common display referenceframe by constructing a destination view for each preprocessed image ofthe plurality of preprocessed images to produce a plurality ofdisplayable images, arranging the plurality of displayable images into adisplay pattern for viewing on the display screen, displaying theplurality of displayable images on a display screen, and accepting usergestures to dynamically alter the common display reference frame.

In yet other embodiments, the operations may further includetranslating, rotating, and zooming in and out of the plurality of imagesin unison on the display screen in response to an input gesture from aninterface device to provide a desired perspective of the imagedbiological tissue sample, removing one or more images from the pluralityof images on the display screen to declutter the display screen, addingnew mode images onto the display screen, rearranging the display patternto form an alternative display pattern, stacking two or more image modesto reinforce image features, and saving the display pattern of a currentexamination as a saved template for future examinations.

In one enablement of this patent, collections of pre-registered imagesmight be provided by the FOV analysis. Examples of FOV analysis aredescribed herein. The images are tagged with metadata describing theirindividual placement, rotation, and magnification, with respect to acommon frame of reference. Together with any new reference frame, themetadata may define an affine mapping between original reference frameof the image and the new frame.

Reimaging to the new frame may be accomplished by mapping a destinationpixel in the new frame back to its corresponding location in the sourceframe of an image, and choosing that pixel value, or a an interpolationof surrounding source pixel values, as the destination pixel value. Inthis way, any image can be translated, rotated, stretched, or shrunk tothe new reference frame shared by all other images in preparation forsimultaneous display.

Deciding which arrangements are important for a diagnostician may bebased entirely on the best judgement of the expert reader. Some viewsmay be deemed unimportant for the case at hand, while still others mightbe added to the collection as being more important for diagnosis.

Embodiments of the present invention are particularly advantageous as anautomatic and reliable technique is provided to identify fields of viewin a multi-channel image while avoiding the tedious effort of manuallymarking fields of view in a multi-channel image by a pathologist orbiologist and thereby also eliminating subjective judgment and humanerror. As the spatial low pass filtering, the local maximum filteringand thresholding operations can be executed at high processing speeds,the computational expense and the latency time experienced by the usercan be minimized. This is due to the fact that the definition of thefields of view is not performed directly on the multi-channel image buton the basis of the filtered and thresholded image which enables thehigh processing speed.

It is to be noted that the analysis in step f is executed on the fullresolution multi-channel image and not on the spatial low pass filteredunmixed image. This assures that the full amount of the availablepictorial information can be used for performing the analysis while thefiltering operation, namely steps b, c and d, merely serve foridentification of the relevant fields of view where a full analysis isto be performed.

In accordance with a further embodiment of the invention one of theunmixed images is processed for defining the field of view as describedabove while another one of the unmixed images is segmented foridentification of tissue regions. The unmixed image can be generatedfrom a single stain image (2-channel, e.g. embodiment of FIG. 2 with astain and a counter-stain) or from a multiplex image (more than 2channels).

Suitable segmentation techniques are as such known from the prior art,(cf. Digital Image Processing, Third Edition, Rafael C. Gonzalez,Richard E. Woods, chapter 10, page 689 and Handbook of Medical Imaging,Processing and Analysis, Isaac N. Bankman, Academic Press, 2000, chapter2). By means of the segmentation non-tissue regions are removed as thenon-tissue regions are not of interest for the analysis.

The segmentation provides a mask by which those non-tissue regions areremoved. The resultant tissue mask can be applied onto the unmixed imageprior or after the spatial low pass or local maximum filtering orthresholding operations and before or after the fields of view aredefined. It may be advantageous to apply the tissue mask at an earlystage in order to further reduce the processing load, such as before theexecution of the spatial low pass filtering.

In accordance with an embodiment of the invention the other one of theunmixed images that is segmented for providing the tissue mask isobtained from the channel that is representative of one stain that is acounter-stain to the stain represented by the unmixed image that isprocessed in accordance with steps b-e of claim 1.

In accordance with an embodiment of the invention fields of view aredefined for at least two of the unmixed images. Fields of view that aredefined in two different unmixed images can be merged if they arelocated at the same or almost identical image location. This isparticularly advantageous for stains that can be co-located such that asingle field of view results for the co-located stains that identify acommon biological structure. By merging such fields of view theprocessing load is further reduced and the analysis in step f needs onlyto be performed once for the merged field of view. Moreover, thecognitive burden for the pathologist or biologist is also reduced asonly one analysis result is presented rather than two related results.Depending on the implementation, the two fields of view may be merged ifa degree of spatial overlap of the fields of view is above an overlapthreshold.

In accordance with embodiments of the invention the analysis of thefield of view is performed by cell counting of the biological cellsshown in the multi-channel image within the considered field of view.The cell counting can be performed by using a suitable image analysistechnique which is applied on the field of view. In particular, the cellcounting can be executed by means of an image classification technique.

In accordance with further embodiments of the invention the analysis ofthe field of view is performed by means of a trained convolutionalneural network such as by entering the field of view or an image patchtaken from the field of view into the convolutional neural network fordetermining a probability for the presence of a biological featurewithin the field of view or the image patch, respectively. An imagepatch may be extracted from the field of view for entry into theconvolutional neural network by first identifying an object of interestwithin the field of view and then extracting the image patch thatcontains this object of interest.

In accordance with a further embodiment of the invention the analysis isperformed on the field of view in step f as a data analysis, such as acluster analysis or statistical analysis.

In accordance with another aspect of the invention an image processingsystem for analyzing a multi-channel image obtained from a biologicaltissue sample being stained by multiple stains is provided that isconfigured to execute a method of the invention.

The subject disclosure features preprocessing systems and methods forautomatic field of view (FOV) selection based on a density of each cellmarker in a whole slide image. Operations described herein includereading images for individual markers from an unmixed multiplex slide orfrom singularly stained slides, and computing the tissue region maskfrom the individual marker image.

A heat map of each marker may be determined by applying a low passfilter on an individual marker image channel, and selecting the top Khighest intensity regions from the heat map as the candidate FOVs foreach marker. The candidate FOVs from the individual marker images aremerged together. The merging may comprise one or both of adding all ofthe FOVs together in the same coordinate system, or only adding the FOVsfrom the selected marker images, based on an input preference or choice,by first registering all the individual marker images to a commoncoordinate system and merging through morphologic operations. Afterthat, all of the identified FOVs are transferred back to the originalimages using inverse registration to obtain the corresponding FOV imageat high resolution.

In some embodiments, lower-resolution images are used to speedcomputation of the FOVs. Because the images are lower resolution, it iscomputationally much faster to compute the heat map and tissue regionmask. This allows the selection of the FOVs to be made automatic andrapid, which allows for faster analysis of the tissue sample.

Tissue slide images contain many features, only some of which are ofinterest for any particular study. Those interesting regions may have aspecific color brought about by selective stain uptake. They may alsohave broad spatial extent. Importantly, the uninteresting regions mayhave some specific spatial frequencies that enable their removal from animage by way of spatial frequency filtering. Such filters include, butare not limited to, low pass, high pass, and band pass, filters. Morecarefully tuned spatial frequency filters may be those known as matchedfilters. Non-limiting examples of spatial frequency filters include, butare not limited to, low pass filters, high-pass filters, band-passfilters, multiple-passband filters, and matched filters. Such filtersmay be statically defined, or adaptively generated.

In the process of locating regions of interest, it is therefore helpfulto first select the proper color by an unmixing process, which can beviewed as a linear operator applied to the primary color channels, R, G,and B, of the image. Spatial frequency filtering is also applied to givepreference to features of interest in the image. These operations may beapplied in either order since they are both linear operators.

In parallel with this region selection, there may be a broadersegmentation mask formed by using entirely differently tuned spatialfrequency filters, to select, for example, only the gross region of theslide image where tissue resides, and rejecting empty regions.Therefore, multiple different spatial frequency filters may be appliedto the same tissue slide image.

Once filtered, a region of interest may be located by applying a localmax filter, a kind of morphological nonlinear filter, which produces animage by making each pixel of the result hold the value of the maximumpixel value from the source image that lies beneath the kernel of themax filter. The kernel is a geometric mask of arbitrary shape and size,but would be constructed for this purpose to have dimensions on theorder of the interesting features. The output image from a local maxfilter will tend to have islands shaped like the kernel and withconstant values equal to the maximum pixel value in that region.

In some embodiments, with the present construction of a local max filterimage, a threshold may be applied to convert the filter image to abinary mask, by assigning binary mask values of 1 to correspondingfilter image pixels above the threshold, and values of 0 tocorresponding filter image pixels below the threshold. The result willbe blobs of 1's that can be labeled as regions, and with measureablespatial extents. Together, these region labels, locations, and spatialextents provide a record of regions of interest (ROIs), or fields ofview (FOVs).

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1B respectively depict a system and a workflow for automaticFOV selection, according to an exemplary embodiment of the presentsubject disclosure.

FIG. 2 depicts a heat map computation, according to an exemplaryembodiment of the present subject disclosure.

FIG. 3 depicts a tissue mask computation, according to an exemplaryembodiment of the subject disclosure.

FIG. 4 depicts candidate FOVs, according to an exemplary embodiment ofthe subject disclosure.

FIGS. 5A-5B depict merging of FOVs from all markers and from selectedmarkers, respectively, according to an exemplary embodiment of thesubject disclosure.

FIGS. 6A-6C depict integrating FOVs, according to exemplary embodimentsof the subject disclosure.

FIG. 7 depicts a user interface for image analysis using an all markerview, according to an exemplary embodiment of the subject disclosure.

FIG. 8 depicts a user interface for image analysis using an individualmarker view, according to an exemplary embodiment of the subjectdisclosure.

FIG. 9 depicts a digital pathology workflow for immunoscore computation,according to an exemplary embodiment of the subject disclosure.

FIG. 10 depicts a process flow chart for an exemplary embodiment of thepresent invention.

FIGS. 11a and 11b depicts a process flow chart for an exemplaryembodiment of the present invention starting with single-stain markerimages.

FIG. 12 depicts a process flow chart for an exemplary embodiment of thepresent invention starting with a multiplex slide.

FIG. 13 depicts a process flow chart for an exemplary embodiment of thepresent invention starting with a single stain image.

FIG. 14 depicts an exemplary process flow chart for simultaneouslydisplaying multiple views according to an embodiment of the presentinvention.

FIG. 15 depicts an exemplary process flow chart for choosing a commondisplay reference frame according to an embodiment of the presentinvention

FIG. 16 depicts an exemplary process flow chart for convertingpreprocessed images to produce displayable images according to anembodiment of the present invention.

FIG. 17 depicts a translated view of images on a user interfaceaccording to an exemplary embodiment of the subject disclosure.

FIG. 18 depicts a rotated view of images on a user interface accordingto an exemplary embodiment of the subject disclosure.

FIG. 19 depicts two images deleted from a user interface according to anexemplary embodiment of the subject disclosure.

FIG. 20 depicts a rearranged display pattern of images on a userinterface according to an exemplary embodiment of the subjectdisclosure.

FIG. 21 depicts a zoomed in view of an image on a user interfaceaccording to an exemplary embodiment of the subject disclosure.

FIG. 22 depicts a stacked view of two images on a user interfaceaccording to an exemplary embodiment of the subject disclosure.

FIG. 23 depicts a schematic diagram illustrating embodiments of thepresent invention.

FIG. 24 illustrates an embodiment of the present invention where a pinchgesture is used to zoom in or zoom out.

DETAILED DESCRIPTION OF THE SUBJECT DISCLOSURE

The present invention features a system and method of simultaneouslydisplaying multiple views of a same region of a biological specimen, forexample, a tissue sample. In some embodiments, the system may comprise aprocessor and a memory coupled to the processor. The memory can storecomputer-readable instructions that, when executed by the processor,cause the processor to perform operations.

In other embodiments, the method may be implemented by an imaginganalysis system and may be stored on a computer-readable medium. Themethod may comprise logical instructions that are executed by aprocessor to perform operations.

As shown in FIG. 14, operations for the system and method describedherein can include, but are not limited to, receiving a plurality ofpreprocessed images of the biological tissue sample (2100), choosing acommon display reference frame that is used for image visualization(2110), converting the plurality of preprocessed images to the commondisplay reference frame by constructing a destination view for eachpreprocessed image of the plurality of preprocessed images to produce aplurality of displayable images (2120), arranging the plurality ofdisplayable images into a display pattern for viewing on the displayscreen (2130), displaying the plurality of displayable images on adisplay screen (2140), and accepting user gestures to dynamically alterthe common display reference frame (2150). Without wishing to limit thepresent invention to a particular theory or mechanism, the presentinvention allows for a coordinated review of the plurality of imagesthat are shown adjacent to one another on a single viewing screen.

In some embodiments, displaying of the plurality of displayable images(2140) may allow for simultaneous dynamic viewing of different aspectsof the imaged biological tissue sample. Repeating the conversion process(2120) may cause all displayable images to simultaneously performapparent coordinated translation, rotation, or magnification changes.

In some embodiments, each preprocessed image may show a view mode of asame region of the biological tissue sample, and each preprocessed imagemay have metadata that describe an image reference frame with respect toa global standard reference frame. The metadata of each preprocessedimage may describe a preprocessed image local reference frame (PI-LRF)with respect to a global standard reference frame (GSRF). For example,the metadata may describe the spatial location, orientation, andmagnification of the preprocessed image with respect to the globalstandard reference frame. As another example, the metadata describestranslation, rotation, and magnification of each image with respect to astandard reference frame. By knowing the common display reference frame,an affine transformation is created to associate source image pixels todisplayed pixels for an image mode view. As used herein, an affinetransformation or, alternatively, an affine mapping, can be defined as alinear transform, expressible as a matrix operator against augmentedposition vectors, which can express arbitrary translations, rotations,and magnifications, of those vectors. Affine transformations are knownto one of ordinary skill in the art.

In some embodiments, the preprocessed image local reference frame(PI-LRF) is a two-dimensional reference frame used to describe alocation of a pixel in the preprocessed image.

In other embodiments, the global standard reference frame is anagreed-upon, fixed two-dimensional reference frame used to describe aspace of pixel locations and which allows an understanding of spatialrelationships between different images by defining affine mappingsbetween each image local reference frame (I-LRF) and the global standardreference frame. In some embodiments, the metadata of each preprocessedimage describe the spatial location, orientation, and magnification ofthe preprocessed image with respect to the GSRF. For example, themetadata can define a first affine mapping between the image referenceframe and the global standard reference frame.

In some embodiments, as shown in FIG. 15, the operation of choosing acommon display reference frame (2110) may further comprise creating atwo-dimensional display image pixel grid (2111), constructing atwo-dimensional display image local reference frame (DI-LRF) used todescribe pixel locations in the display image pixel grid (2112),choosing a location, orientation, and magnification for the DI-LRF withrespect to the GSRF (2113), and computing an affine transform that mapspixel locations in the DI-LRF to locations in the GSRF (2114). The gridintersections can denote pixel locations. This construction can servesas a display image template and may provide an affine partial mappingfor production of display images.

In some embodiments, as shown in FIG. 16, the operation of convertingthe plurality of preprocessed images to the common display referenceframe (2120) may further comprise constructing a working copy of theCDRF display image template and affine partial mapping (2121), composingthe affine partial mapping with the first affine mapping for thepreprocessed image to produce a composite mapping that transforms pixellocations in the DI-LRF of the display image to a location in the PI-LRFof the preprocessed image (2122), and painting the display image byperforming operations for each display image pixel (2123). In someembodiments, the working copy of the display image template comprisesmemory cells to hold pixel values for a display image.

Operations for painting the display image may include, but are notlimited to, mapping with the composite affine transform from a DI-LRFlocation of the display image pixel to a location in the PI-LRF of thepreprocessed image (2124), interpolating a pixel value among neighboringpixels in the preprocessed image around that mapped location (2125), anddelivering the interpolated pixel value as the pixel value used in thedisplay image at the display image pixel (2126). By performing theseoperations for each display image pixel, each preprocessed image may betransformed to the display image for representation on the displayscreen.

In some embodiments, interpolation among neighboring pixels (2125) maybe performed by simply choosing the nearest pixel for its value, or byusing bilinear interpolation among the four nearest neighboring pixels.In other embodiments, when magnification is changed between source andtarget images, more elaborate methods, such as spatial low-passfiltering, may be required to avoid sample aliasing or imagingartifacts, since this is equivalent to sample rate conversion.

In other embodiments, the operation of converting the plurality ofpreprocessed images (2120) may perform nonlinear corrections on theplurality of preprocessed images to remove optical distortions.Exemplary nonlinear corrections may include removal of pincushion orbarrel distortion, defocus, coma, or astigmatism.

In some embodiments, any of the two-dimensional reference frames asmentioned herein, such as the two-dimensional local reference frames(PI-LRFs and the DI-LRF) and the agreed-upon fixed two-dimensionalreference frame (GSRF), can be orthogonal Cartesian reference frames. Inother embodiments, any of the two-dimensional reference frames asmentioned herein can be non-orthogonal and/or non-Cartesian referenceframes.

In some embodiments, the plurality of images is produced bypreprocessing images of the biological tissue sample. Preprocessing ofthe images may utilize methods such as the FOV methods as describedherein. However, it is understood that other suitable methods may beused to preprocess the images.

In some embodiments, the display pattern may be in the form of rows andcolumns. This display pattern may feature an “m” number of rows and an“n” number of columns, where “m” and “n” can be any natural number. Forexample, the display pattern may have 2 rows and 3 columns. In otherembodiments, the display pattern may be a ring or a square. In stillother embodiments, the display pattern may be a pyramid.

In other embodiment, the operations may further comprise translating theplurality of images in unison on the display screen in response to aninput gesture from an interface device, rotating the plurality of imagesin unison on the display screen in response to an input gesture from aninterface device, and zooming in and out of the plurality of images inunison on the display screen in response to an input gesture from aninterface device. As shown in FIGS. 17-19, the operations oftranslating, rotating, and zooming of the plurality of images mayprovide a desired perspective of the imaged biological tissue sample.For example, translating of the plurality of images may involve slidingthe images in a linear direction. Rotation of the plurality of imagesmay be performed in a clockwise or counterclockwise direction. Zoomingin on the plurality of images may provide for a closer view of a regionof the biological tissue sample. Zooming out of the plurality of imagesmay provide for a distant view of the biological tissue sample.

In some embodiments, as shown in FIG. 20 the operations may furthercomprise removing one or more images from the plurality of images on thedisplay screen to declutter the display screen. For example, if an imageshows an undesirable or irrelevant view of the biological tissue sample,the image may be removed. In other embodiments, the operations mayfurther comprise adding new mode images onto the display screen. The newmode images may be viewed in tandem with other image modes.

Non-limiting examples of modes in which images may be viewed can includea variety of color channels, image filter states, or edge detectionstates. Generally, there may be useful alterations of an original imagethat highlight certain characteristics, which could offer simultaneousviews containing important features of diagnostic interest to the expertreader.

In some embodiments, as shown in FIG. 21, the operations may furthercomprise rearranging the display pattern to form an alternative displaypattern. The alternative display pattern may bring together image modesfor closer inspection. In other embodiments, as shown in FIG. 22, theoperations may further comprise stacking two or more image modes toreinforce image features. Stacking of the two or more image modes can bein response to an input gesture from an interface device. In someembodiments, the two or more image modes may be translucent.

In other embodiments, the operations may further comprise saving thedisplay pattern of a current examination as a saved template tofacilitate displaying of another plurality of images in futureexaminations.

In one embodiment of this invention, the expert reader can affect allimages simultaneously by invoking actions on only one of the images suchthat all images respond in tandem. Non-limiting exemplary input gesturesand interface devices may include, but are not limited to, a mouse, ahaptic sensor, eye sensors, and electronic cameras. For example, anexpert reader might use a mouse click to activate one of the images, andthen rotate the mouse wheel to affect zoom magnification of the images.Mouse click and drag within an activated image might drag all images inthe same direction. As another example, a haptic sensor might be used toperform selected image changes. The haptic sensor may offer rotation,translation, zooming, stacking, etc, which may be more elaborate than asimple computer mouse.

Eye sensors can detect eye gestures of the expert reader, such aschanging the center of sight attention, blinking, etc. Electroniccameras can witness special gestures of an operator, such as handmotion, that indicate image translation, rotation, magnification,display rearrangement, image stacking, and control of translucenceduring stacking, etc. In other embodiments, any sufficient and validmanner of interacting with a device, such as a computer, may be used,with a preference for the simplest and most direct interaction toachieve the expert reader's aims.

In alternative embodiments, the method of simultaneously displayingmultiple views of a same region may be used in examination ofmultispectral Earth surface imagery for remote sensing applications, orfor battlefield management.

A non-limiting example of implementing the method of simultaneouslydisplaying multiple views of a same region of a biological tissue sampleon a display screen may feature:

1. Loading data for the biological tissue sample.2. Selecting a file from a file list.3. Displaying six images from the selected file in a display pattern of3 columns by 2 rows.4. Selecting important markers.5. Displaying a heat map for a marker of the image sample.6. Switching between an original view, a heat map view, or an individualmarker view.7. Displaying hot spots of the image sample.8. Aligning to a same coordinate system.9. Rotating, translating, or zooming in and out of the images.

10. Merging the FOVs.

11. Assigning a label to a region of the imaged sample.12. Renaming an image.13. Adding or deleting images.14. Saving the file.

Preprocessing of Images

In some embodiments, the present invention may utilize systems andmethods for preprocessing of biological slide images. It is understoodthat any suitable system or method may be used to preprocess the images.In one embodiment, a non-limiting example of a preprocessing system ormethod may feature an automatic field of view (FOV) selection based on adensity of each cell marker in a whole slide image. Operations describedherein include, but are not limited to, reading images for individualmarkers from an unmixed multiplex slide or from singularly stainedslides, and computing the tissue region mask from the individual markerimage. A heat map of each marker may be determined by applying a lowpass filter on an individual marker image channel, and selecting the topK highest intensity regions from the heat map as the candidate FOVs foreach marker. The candidate FOVs from the individual marker images maythen be merged together. The merging may comprise one or both of addingall of the FOVs together in the same coordinate system, or only addingthe FOVs from the selected marker images, based on an input preferenceor choice, by first registering all the individual marker images to acommon coordinate system and merging through morphologic operations.Subsequently, all of the identified FOVs are transferred back to theoriginal images using inverse registration to obtain the correspondingFOV image at high resolution. Without wishing to limit the presentinvention to any theory or mechanism, the systems and methods of thepresent invention may offer advantages such as being reproducible,unbiased to human readers, and more efficient.

In some embodiments, the system for quality control of automatedwhole-slide analysis comprises an image acquisition system (102), aprocessor (105); and a memory coupled to the processor (110). The memoryis configured to store computer-readable instructions that, whenexecuted by the processor, cause the processor to perform operations oneor more of the following operations (but not limited to the followingoperations) comprising: reading a high resolution input image (231) fromthe image acquisition system (102), computing a low resolution versionof the high resolution input image, reading a plurality of lowresolution image marker images from the image acquisition system (102),wherein each image marker image is of a single color channel (232) ofthe low resolution input image, computing a tissue region mask (233)corresponding to the low resolution input image, computing a low passfiltered image (234) of each image marker image (114), generating amasked filtered for each image marker image (113), where the maskedfiltered image is the tissue region mask multiplied by the low passfiltered image, identifying a plurality of candidate fields of view(FOVs) within each masked filtered image (116), merging a subset of aplurality of candidate FOVs for each image marker image (117), into aplurality of merged FOVs, and depicting the merged portion of theplurality of candidate fields of view on the input image.

In some embodiments, a heat map may be computed for the masked filteredimage. In some embodiments, the heat map comprises applying colors tothe masked filtered image, wherein low intensity regions are assigned toblue colors and higher intensity regions are assigned to yellow orangeand red colors. Any other appropriate colors or combinations of colorsmay be used to assign low and high intensity regions.

In some embodiments, the generation of the tissue region mask comprisesone or more of the following operations (but not limited to thefollowing operations): computing the luminance (337) of the lowresolution input image (336), producing a luminance image (338),applying a standard deviation filter to the luminance image (339),producing a filtered luminance image (340), and applying a threshold tofiltered luminance image (341), such that pixels with a luminance abovea given threshold are set to one, and pixels below the threshold are setto zero, producing the tissue region mask (342).

In some embodiments, the tissue region mask is computed directly fromthe high resolution input image. In this case, the tissue region maskmay be converted to a lower resolution image before application to thefiltered image market images.

In some embodiments, the image marker images are obtained by unmixing(111) a multiplex slide, where the unmixing module uses a referencecolor matrix (112) to determine what colors correspond to the individualcolor channels. In other embodiments, the image marker images areobtained from single stain slides.

In some embodiments, the image registration process comprises selectingone image marker image to serve as a reference image, and computing atransformation of each image marker to the coordinate frame of thereference image. The methods for computing a transformation of eachimage to a reference image are well known to those skilled in the art.In other embodiments, if the images are obtained by unmixing a multiplexreference slide, no registration is needed since all the unmixed imagesare already in the same coordinate system.

The subject disclosure provides systems and methods for automatic fieldof view (FOV) selection. In some embodiments, the FOV selection is basedon a density of each cell marker in a whole slide image. Operationsdescribed herein include reading images for individual markers from anunmixed multiplex slide or from singularly stained slides, and computingthe tissue region mask from the individual marker image. A maskedfiltered image of each marker may be determined by applying a low passfilter on an individual marker image channel, and applying the tissueregion mask. The top K highest intensity regions from the maskedfiltered image are selected as the candidate FOVs for each marker. Thecandidate FOVs from the individual marker images are merged together.The merging may comprise one or both of adding all of the FOVs togetherin the same coordinate system, or only adding the FOVs from the selectedmarker images, based on an input preference or choice, by firstregistering all the individual marker images to a common coordinatesystem and merging through morphologic operations. After that, all ofthe identified FOVs are transferred back to the original images usinginverse registration to obtain the corresponding FOV image at highresolution. Without wishing to limit the present invention to any theoryor mechanism, the systems and methods of the present invention may offeradvantages such as being reproducible, unbiased to human readers, andmore efficient. As a result, a digital pathology workflow for automaticFOV selection, in accordance with the subject disclosure, includes acomputer-based FOV selection algorithm that automatically provides thecandidate FOVs that may be further analyzed by a pathologist or otherevaluator.

The operations described herein have been described, for exemplarypurposes, in connection with the identification of immune cells, and foruse in immunoscore computations. However, the systems and methods may beapplicable to any type of image of a cell or biological specimen, andare applicable to determinations of type, density and location for anytype of cell or group of cells. As used herein, the terms “biologicalspecimen” and “biological tissue sample” may be used interchangeably.Moreover, besides cancerous tissue and immune markers, the subjectdisclosure is applicable to any biological specimen or tumor of anydisease or non-disease state, and images of biological specimens thathave been subjected to any type of staining, such as images ofbiological specimens that have been stained with fluorescent andnon-fluorescent stains. Also, one of ordinary skill in the art wouldrecognize that the order of the steps may vary from what is describedherein.

FIGS. 1A-1B respectively depict a system 100 and a workflow forautomatic FOV selection, according to an exemplary embodiment of thepresent subject disclosure. Referring to FIG. 1A, a system 100 comprisesa memory 110, which stores a plurality of processing modules or logicalinstructions that are executed by processor 105 coupled to computer 101.An input from image acquisition system 102 may trigger the execution ofone or more of the plurality of processing modules. Besides processor105 and memory 110, computer 101 also includes user input and outputdevices such as a keyboard, mouse, stylus, and a display/touchscreen. Aswill be explained in the following discussion, processor 105 executeslogical instructions stored on memory 110, including automaticallyidentifying one or more FOVs in an image of a slide (containing abiological specimen, such as a tissue sample) that has been stained withone or more stains (for example, fluorophores, quantum dots, reagents,tyramides, DAPI, etc.).

Image acquisition system 102 may include a detector system, such as aCCD detection system, or a scanner or camera such as a spectral camera,or a camera on a microscope or a whole-slide scanner having a microscopeand/or imaging components (the image acquisition system is not limitedto the aforementioned examples). For example, a scanner may scan thebiological specimen (which may be placed on a substrate such as aslide), and the image may be saved in a memory of the system as adigitized image. Input information received from image acquisitionsystem 102 may include information about a target tissue type or object,as well as an identification of a staining and/or imaging platform. Forinstance, the sample may have been stained by means of application of astaining assay containing one or more different biomarkers associatedwith chromogenic stains for brightfield imaging or fluorophores forfluorescence imaging. Staining assays can use chromogenic stains forbrightfield imaging, organic fluorophores, quantum dots, or organicfluorophores together with quantum dots for fluorescence imaging, or anyother combination of stains, biomarkers, and viewing or imaging devices.Moreover, a typical sample is processed in an automated staining/assayplatform that applies a staining assay to the sample, resulting in astained sample. Input information may further include which and how manyspecific antibody molecules bind to certain binding sites or targets onthe tissue, such as a tumor marker or a biomarker of specific immunecells. The choice of biomarkers and/or targets may be input into thesystem, enabling a determination of an optimal combination of stains tobe applied to the assay. Additional information input into system 100may include any information related to the staining platform, includinga concentration of chemicals used in staining, a reaction times forchemicals applied to the tissue in staining, and/or pre-analyticconditions of the tissue, such as a tissue age, a fixation method, aduration, how the sample was embedded, cut, etc. Image data and otherinput information may be transmitted directly or may be provided via anetwork, or via a user operating computer 101.

An unmixing module 111 may be executed to unmix the image, for instanceif the image is a multiplex image. Unmixing module 111 unmixes the imageinto individual marker color channels. Unmixing module 111 may read froma reference color matrix database 112 to obtain the reference colormatrix and use the reference color matrix to perform unmixingoperations. If the image is of a single stain slide, the image can bedirectly used for FOV selection. In either case, a heat map computationmodule 113 may be executed to evaluate a heat map for each individualmarker image, or single stain image. A heat map maps the density ofvarious structures or biomarkers on the whole-slide image. To accomplishthis, heat map computation module 113 may perform operations such asassigning colors to a low pass filtered image that is processed by lowpass filter module 114. A tissue region mask may also be applied to thelow pass filtered image. The heat map illustrates pixels according tothe respective densities of the pixels, and thus, corresponds to thedensity of the cell distribution in each image. For example, the heatmap will distinguish high-density pixels from low-density pixels byillustrating higher density pixels in a color that is warmer than acolor used for lower density pixels. Local max filter module 115 may beexecuted to apply a local max filter to the low pass filtered image toobtain the local maxima of the image. Subsequently, a top K FOVselection module 116 may be executed to select the top K regions withthe highest densities from the local max filtered image. The top Kregions are designated as the candidate FOVs for each image. Forexample, the cells may be clustered together in the high-density regionwhile they are more scattered in the low-density region. The FOVs fromeach image are merged together by merge FOV module 117, which performsoperations such as taking all the FOVs or the FOVs from selected markersonly and merging them. A registration module 118 is invoked to transferall the images to the same coordinate system, so that the coordinates ofthe FOVs can be directly added up in the same coordinate system.

As described above, the modules include logic that is executed byprocessor 105. “Logic”, as used herein and throughout this disclosure,refers to any information having the form of instruction signals and/ordata that may be applied to affect the operation of a processor.Software is one example of such logic. Examples of processors arecomputer processors (processing units), microprocessors, digital signalprocessors, controllers and microcontrollers, etc. Logic may be formedfrom signals stored on a computer-readable medium such as memory 110that, in an exemplary embodiment, may be a random access memory (RAM),read-only memories (ROM), erasable/electrically erasable programmableread-only memories (EPROMS/EEPROMS), flash memories, etc. Logic may alsocomprise digital and/or analog hardware circuits, for example, hardwarecircuits comprising logical AND, OR, XOR, NAND, NOR, and other logicaloperations. Logic may be formed from combinations of software andhardware. On a network, logic may be programmed on a server, or acomplex of servers. A particular logic unit is not limited to a singlelogical location on the network. Moreover, the modules need not beexecuted in any specific order. Each module may call another module whenneeded to be executed.

An exemplary workflow for FOV selection is depicted in FIG. 1B. In FIG.1B, N represents the number of markers applied to the slides. For amultiplex slide 121, color unmixing 122 is performed, for exampleaccording to the unmixing method disclosed in Patent Application61/830,620, filed Jun. 3, 2013, and WO 2014/195193 A1 entitled “ImageAdaptive Physiologically Plausible Color Separation”, the disclosure ofwhich is hereby incorporated by reference in its entirety. The methoddisclosed in Patent Application 61/943,265, filed Feb. 21, 2014, andentitled. “Group Sparsity Model for Image Unmixing”, andPCT/EP2014/078392 filed 18 Dec. 2014 which is hereby incorporated byreference in its entirety, is, in an exemplary embodiment utilized toobtain an image 123 for each marker. Otherwise, if the image is a singlestain slide, scanned images 124 of single stain slides for each markerare utilized as an input to an automatic FOV selection system, such asthe system depicted in FIG. 1A. For example, a heat map computationoperation may be performed to compute the hotspot 125 from the image ofeach marker to generate the top candidate FOVs 126 for each marker. Thecandidate FOVs 126 may be integrated 127 to generate the final FOV list128. Final FOV list 128 comprises a list of possible FOVs for selectionby a pathologist to utilize for evaluating the biological specimen, forexample, immune cells.

As used herein and throughout this disclosure, hotspots are regionscontaining a high density of marked (i.e., stained) cells, for examplehotspots can be cells from different types of images and markers such asISH, IHC, fluorescent, quantum dots etc. The subject disclosure usesimmune cells in an IHC image as an example to demonstrate this feature(as previously discussed, the present invention is not limited to immunecells in an IHC image). In light of the subject disclosure, variousalgorithms may be used by those having ordinary skill in the art to findhotspots and to use automatic hotspot selection as a module inimmunoscore computation. Exemplary embodiments of the subject disclosureutilize the automatic FOV selection operations described herein to solvethe problem of avoiding biased manually selected FOVs. To automaticallyidentify FOVs that may be of interest to a pathologist or otherevaluator, a heat map is computed for each marker or image representinga single marker, based on a low-resolution image (e.g. a 5× zoom image).

FIG. 2 depicts a heat map computation, according to an exemplaryembodiment of the present subject disclosure. The operations describedin FIG. 2 illustrate how a heat map computation is utilized to identifyhotspots. For example, given a single-marker channel 232 of an inputimage 231, a low-pass-filtered image 234 is used to generate heat map235, which basically takes the low pass filtered image 234 as input andapplies a color map on top of it for visualization purposes. Forexample, a red color may correspond to high intensity pixels in the lowpass filtered image and a blue color may correspond to low intensitypixels. Other depictions of color and/or intensity may be evident tothose having ordinary skill in the art in light of this disclosure. Atissue region mask 233 may be created by identifying the tissue regionsand excluding the background regions. This identification may be enabledby image analysis operations such as edge detection, etc. Tissue regionmask 233 is used to remove the non-tissue background noise in the image,for example the non-tissue regions.

In the embodiment considered with respect to FIG. 2 the input image 231is stained by means of a stain and its respective counter-stain whichprovides two channels, namely the FP3 channel and the HTX channel. Thetwo-channel image 231 is unmixed which provides the unmixed images 232and 238 of the FP3 and HTX channels, respectively.

The unmixed image 232 is then low pass filtered by means of a spatiallow pass filter which provides the low pass filtered image 234. Next,the heat map 235 may be added to the low pass filtered image 234 forvisualization purposes.

The unmixed image 238 is then used to compute the tissue region mask 233by the method described in FIG. 3.

The low pass filtered image 234 with or without the added heat map 235is then local maximum filtered which provides the local max filteredimage 236. The local max filtered image 236 comprises a number of localmaxima 239, in the example considered here five local maxima 239.1-239.5as depicted in FIG. 2. Next, a thresholding operation is performed onthe local max filtered image 236 such as by applying a threshold ontothe local max filtered image 236 such that only the local maxima 239.1and 239.4 that surpass this threshold are not removed by thethresholding operation.

Alternatively the local maxima 239 are ranked in a sorted list and onlya number of the K topmost local maxima are taken from the list, where Kis 2 for explanatory purposes in the embodiment considered here,resulting in the local maxima 239.1 and 239.4. Each of the local maxima239 consists of a set of neighboring pixels.

This thresholding operation provides the thresholded image 237. Each ofthe local maxima 239.1 and 239.4 in the thresholded image 237 may definethe location of a respective field of view 240.1 and 240.2,respectively. Depending on the implementation, these fields of view240.1 and 240.2 may be candidate fields of view for testing whetherthese fields of view can be merged with other fields of view insubsequent processing operations as described below with respect to FIG.6. The positions of the fields of view 240.1 and 240.2 are defined bymeans of the thresholded image 237 and its local maxima. However, thecontent of the fields of view is taken from the respective image areawithin the original multi-channel image 231 in order to take advantageof the full pictorial information content for performing an imageanalysis of the respective field of view.

FIG. 3 depicts a tissue mask computation, according to an exemplaryembodiment of the subject disclosure, such as to compute tissue mask 233from unmixed image 238 by means of a segmentation technique. A linearcombination 337 of the RGB channels 336 of the tissue RGB image iscomputed to create a grayscale luminance image 338. The combinationweights for the R, G and B channels (e.g. 0.3, 0.6, 0.1 in 337) aresubject to change based on different applications. A 3 pixel by 3 pixelstandard deviation filter 339 is applied to the luminance image 338,resulting in a filtered luminance image 340. Here the filter size (e.g.3 by 3, 5 by 5) is subject to change based on different applications.The tissue mask 342 is a binary image obtained from thresholding 341 thefiltered luminance image 340. For example, tissue mask 342 may compriseregions with pixel intensity value larger than 1.5. The thresholdingparameter MaxLum (e.g. 1.5, 2.0, 3.0) can vary based on differentapplications.

FIG. 4 depicts candidate FOVs, according to an exemplary embodiment ofthe subject disclosure. Candidate FOVs 443 are selected from the top Khighest density regions (also called hot spots) of the heat map. Forexample, K can be chosen from 5, 10, 15, 20 etc. A local maximum filteris applied to the low pass filtered image 234 with the added heat map235 (cf. FIG. 2) in order to provide a local max filtered image. 236 Itis to be noted that the heat map 235 is not essential for the processingbut serves for visualization purposes. A local maximum filter is afunction to identify a constant value connected region of pixels withthe external boundary pixels all having a lower value. It can use 4 or 8connected neighborhoods for 2-D images. The implementation of thisfunctionality is available at Matlab(http://www.mathworks.com/help/images/ref/imregionalmax.html).

The local maximum is obtained as the average intensity with in theconnected region. The local maximum values are sorted providing a sortedlist to produce the rank of the hotspots and top K hotspots are reportedthus thresholding the local max filtered image. Alternatively apredefined threshold is applied on the local maximum filtered image suchthat all hotspots above the threshold are reported. The regions returnedby the local maximum filter computation module are the locations of thelocal maximums.

As described herein, different FOVs may be obtained for different markerimages resulting from unmixing of a multiplex slide or from single stainslides. The FOVs are integrated to ensure that for each patient underdiagnosis, the same set of FOVs is referenced across different markers.There are several possible options to integrate FOVs. FIGS. 5A-5B depictmerging of FOVs from all markers and from selected markers,respectively, according to an exemplary embodiment of the subjectdisclosure. For example, all candidate FOVs from the different markerimages may be merged, as depicted in FIG. 5A. In the alternative,different FOVs for different marker images may be selected and merged,as depicted in FIG. 5B.

Moreover, different FOVs for different marker images may be analyzedindependently based on a user's needs. FIGS. 6A-6B depict integratingFOVs, according to an exemplary embodiment of the subject disclosure.With reference to FIG. 6A, all the FOVs are selected and, with referenceto FIG. 6B, only the FOVs corresponding to specific markers areselected. Each circle 661 represents a possible FOV for the markers.Each dot 662 in each circle 661 represents a local maximum point foreach FOV. Each circle 661 may surround a different marker. Line 663corresponds to the separation between the tumor and the non-tumorregions. FOVs 664 outside of tumor regions are excluded by morphologicaloperations, such as union and intersection. The final FOVs (i.e., theFOVs that are selected for analysis) are the union of all the FOVs fromeach marker, as depicted by the methods of FIGS. 5A and 5B.

In some embodiments, the FOV may be a rectangle about the local maxima.In other embodiments, the FOV may be an arbitrary shape. In someembodiments, the FOV may be a border around a region of high intensity.

FIG. 6B depicts specifying the most important markers for a givenproblem by the user, and merging the FOVs based on the selected markers.For example, assume PF3 and CD8 are the most important markers. All theimages of single markers may be aligned to the same coordinate system(e.g. the reference coordinate can be the slide section in the middle ofthe tissue block or the slide with a specific marker) using imageregistration. Each image may therefore be aligned from its oldcoordinate system to the new reference coordinate system. FOVs ofselected markers (e.g. FP3 and CD8) from an individual marker image maybe aligned to the common space and merged using morphological operationssuch as union and intersection to obtain the merged FOVs (FOVs 665 inFIG. 6B). FIG. 6C shows the morphological operations. Assume A is theFOV from CD8 image and B is the FOV from FP3 image. We first overlay Aand B in the same coordinate system and obtain the overlapped region Cby computing the intersection of A and B. We then evaluate the ratio ofthe area of C and the area of A (or B). If the ratio is greater than athreshold (e.g. 0.6, 0.8, etc.), we select the FOVs, otherwise wediscard the FOVs. The merged FOVs may be mapped back to all the singlemarker images using inverse registration (i.e. align the registeredimage in the new coordinate system back to its original old coordinatesystem) for further analysis. FOVs 664 outside tumor regions areexcluded.

FIGS. 7 and 8 depict user interfaces for image analysis using all markerviews and individual markers views, according to exemplary embodimentsof the subject disclosure. In these exemplary embodiments, a userinterface associated with a computing device may be utilized to performthe FOV selection. The user interface may have All Markerfunctionalities (FIG. 7) and Single Marker Functionalities (FIG. 8). Themarker functions can be accessed by selecting from a tab on the top ofthe user interface. When using the All Marker function as shown in FIG.7, all the markers may be viewed and the heat map computation, FOVselection, key marker selection, registration and inverse registrationcan be performed. In the All Marker View (i.e., a view that illustratesall the markers side by side) options may be provided such as loading alist 771 of image folders (a) with each folder containing all the imagesincluding the multiplex and single stains for the same case. Allow batchprocessing of all the images in the list. Other options provided in afeature panel 772 may include linking the axes for all the images tosimultaneously zoom in and out on the images to view the correspondingregions (b), selecting the number of FOVs (c), align the images to acommon coordinate system (d), and allowing the user to pick the mostimportant markers for integrating FOVs (e). Colors may be depictedindicating the markers that the FOVs come from. Further options providedmay include allowing the user to switch 774 between the heat map viewand IHC view, and computing 773 the heat map of each image.

FIG. 8 depicts the Individual Marker View or Single Marker View,displaying the final selected FOVs for each marker. Features provided inthis view may include displaying a thumbnail 881 of the whole slideimage, with the FOVs annotated by box in the thumbnail image and a textnumber near the box indicating the index of the FOV. Other features mayinclude allowing the user to select from the FOV list 883 to deleteun-wanted FOVs using checkbox, displaying the high resolution image ofthe selected FOV 882, saving the image of each FOV into a local folderat original resolution (d), and allowing the user to assign a label toeach FOV (e). The labels can be the regions associated with the FOV suchas peripheral region, tumor region, and lymphocyte region etc. It willbe recognized by those having ordinary skill in the art that theseexemplary interfaces may differ from application to application andacross various computing technologies, and may use different versions ofinterface so long as the novel features described herein are enabled inlight of this disclosure.

Therefore, the systems and methods disclosed herein provide automaticFOV selection, and have been found important to analyzing biologicalspecimens, and useful in computing tissue analyses scores, for examplein immunoscore computations. Operations disclosed herein overcomedisadvantages known in the prior art, such as FOV selection beingun-reproducible and biased in human reader manual FOV selection, as theautomatic FOV selection is able to provide the FOVs via a computerwithout relying on a human reader's manual selection. When combined withautomatic immune cell counting and data analysis, the disclosedoperations allow a complete automatic workflow that takes in one or morescanned images or image data as input, and outputs the final clinicaloutcome prediction. The systems and methods disclosed herein provideautomatic FOV selection, and have been found important to analyzingbiological specimens, and useful in computing tissue analyses scores,for example in immunoscore computations. Operations disclosed hereinovercome disadvantages known in the prior art, such as FOV selectionbeing un-reproducible and biased in human reader manual FOV selection,as the automatic FOV selection is able to provide the FOVs via acomputer without relying on a human reader's manual selection. Whencombined with automatic immune cell counting and data analysis, thedisclosed operations allow a complete automatic workflow that takes inone or more scanned images or image data as input, and outputs the finalclinical outcome prediction.

FIG. 9 depicts a digital pathology workflow for immunoscore computation,according to an exemplary embodiment of the subject disclosure. Thisembodiment illustrates how the automatic FOV selection method disclosedherein may be utilized in an immunoscore computation workflow. Forexample, after a slide is scanned 991 and the FOVs have been selected992 according to the operations disclosed herein, an automatic detection993 of different types of cells in each FOV can be performed. Theautomatic cell detection technique, for example, according to the methoddisclosed in U.S. Patent Application Ser. No. 62/002,633 filed May 23,2014 and PCT/EP2015/061226, entitled “Deep Learning for Cell Detection”,which is hereby incorporated by reference in its entirety, is anexemplary embodiment utilized to obtain detect the cells. Further,features (e.g., features related to the number and/or types of cellsidentified) can be extracted 994 that are related the one or more cellsdetected for each biological specimen (e.g., tissue samples, etc.). Thefeatures can be number of different types of cells and the ratios ofcells in different FOVs related to different regions in the tissue imagesuch as the tumor region and the periphery region. Those features can beused to train 995 a classifier (such as Random Forest and Support VectorMachine) and classify each case to the different outcome classes (e.g.,likelihood of relapse or not).

FIG. 10 depicts a process flow for an exemplary embodiment of thepresent invention. An input image (1001) is received from the imageacquisition system. In addition, a series of low-resolution markerimages (1004) are received from the image acquisition system. The markerimages may be derived by unmixing of the high-resolution image or may bereceived as single stain slide images. The low resolution input image isused to compute a tissue region mask (1003), which indicates which partsof the image contain tissue of interest. The low resolution image markerimages are passed through a low pass filter to produce filtered imagemarker images (1005). The tissue region mask is then applied to the lowpass filtered images to block out (reduce to 0) regions that are not ofinterest. The results in a masked filtered image (1006) for each marker.A local max filter is applied to a max filtered image to identify localmaxima (1007). The top K local maxima are selected (1008), and for eachlocal maxima a field of view is defined (1009). Then the FOVs for eachimage are merged (1010), by transferring all images to a commoncoordinate frame and overlaying and combining any overlapping fields ofview. The merged fields of view are then transferred back to theoriginal image coordinate system, extracting the regions from the highresolution input image for analysis.

FIG. 11 shows a different process flow for another exemplary embodimentof the present invention. The process flow is divided into a FOVgeneration step (1100) as shown in FIG. 11a , and a field of viewmerging step (1124) as shown in FIG. 11b . In the FOV generation step,single stain images (1101) are received from the image acquisitionsystem. The images are low-pass filtered (1102). In some embodiments,the images may be converted to a lower resolution (1103), which speedsprocessing. In some embodiments an unmixing step (1104) may be appliedto extract the color channel of interest from the single stain slides,if it is not already reduced to a single color channel, producing singlemarker images (1108). In some embodiments an HTX image (1105) may alsobe generated. The single marker image is then segmented (1109) toidentify features of interest. From the segmented image a tissue regionmask (1110) is generated. In some embodiments, the single marker imagemay be visualized (1106) using a heat map (1107), by assigning colors toregions of varying intensity in the single marker image. The tissueregion mask (1110) is then applied to the single marker image (1111),resulting in a foreground image (1112), which displays the intensity ofthe marker image only in the tissue region of interest. The foregroundimage is passed through a local max filter (1113), to identify peaks inintensity. Candidate FOV coordinates are identified as the top K peaksof the local max filtered image (1114). Finally, regions around eachcandidate FOV coordinate are defined (1115) to obtain the list ofcandidate FOVs (1116). These operations are performed for each singlestain slide.

In the FOV merging step (1124), all of the candidate FOV lists for thevarious single stain slides are obtained (1117). The images areregistered to a single coordinate frame (1118), by selecting one imageas a reference image and transforming the other images to match thereference image. The candidate FOV coordinates are then transformedaccordingly to obtain aligned candidate FOV lists (1119). The FOVs arethen overlaid and merged (1120), to obtain a unified FOV list for allimages (1121). Inverse registration is then performed (1122) totransform the unified FOVs back to each of the original coordinatesystems of the original single stain images (1123). The FOVs can then bedisplayed on the original single stain slides.

FIG. 12 shows process flow of an alternative embodiment of the presentinvention, using multiplex slides as inputs (1201). In the FOVgeneration step, multiplex slides (1201) are received from the imageacquisition system. The images are low-pass filtered (1202). In someembodiments, the images may be converted to a lower resolution (1203),which speeds processing. In this embodiment, an unmixing step (1204) isapplied to extract the color channels of interest from the multiplexslide, producing a plurality of single marker images (1208). In someembodiments an HTX image (1205) may also be generated. The first singlemarker image is then segmented (1209) to identify features of interest.From the segmented image a tissue region mask (1210) is generated. Insome embodiments, the single marker image may be visualized (1265) usinga heat map (1207), by assigning colors to regions of varying intensityin the single marker image. The tissue region mask (1210) is thenapplied to the single marker image (1210), resulting in a foregroundimage (1212) which displays the intensity of the marker image only inthe tissue region of interest. The foreground image is passed through alocal max filter (1213), to identify peaks in intensity. Candidate FOVcoordinates are identified as the top K peaks of the local max filteredimage (1214). Finally, regions around each candidate FOV coordinate aredefined (1215) to obtain the list of candidate FOVs (1216). Theseoperations are performed for each single stain slide in order. The FOVmerging step proceeds as in FIG. 11 b.

FIG. 13 shows yet another process flow of an alternative embodiment ofthe present invention, using single stain images (1301) as inputs. Theimages are low-pass filtered (1302). In some embodiments, the images maybe converted to a lower resolution (1303), which speeds processing. Insome embodiments an unmixing step (1304) may be applied to extract thecolor channel of interest from the single stain slides, if it is notalready reduced to a single color channel, producing single markerimages (1308). In some embodiments an HTX image (1305) may also begenerated. In other embodiments, the single marker image may bevisualized (1306) using a heat map (1307), by assigning colors toregions of varying intensity in the single marker image. In oneembodiment, the lower resolution images are segmented (1309) to identifyfeatures of interest. From the segmented image, a tissue region mask(1310) is generated and then the mask operation is applied (1311) to thesegmented image, resulting in a foreground image (1312), which displaysthe intensity of the marker image only in the tissue region of interest.In another embodiment, the mask operation (1311) is applied to thesingle marker image (1308), resulting in a foreground image (1312). Ineither embodiment, the foreground image (1312) is passed through a localmax filter (1313) to identify peaks in intensity. Candidate FOVcoordinates are identified as the top K peaks of the local max filteredimage (1314). Finally, regions around each candidate FOV coordinate aredefined (1315) to obtain the list of candidate FOVs (1316). Theseoperations are performed for each single stain slide. The FOV mergingstep proceeds as in FIG. 11 b.

The computer-implemented method for automatic FOV selection, inaccordance with the present invention, has been described, for exemplarypurposes, in connection with the identification of immune cells, and foruse in immunoscore computations. However, the computer-implementedmethod for automatic FOV selection, in accordance with the presentinvention, is applicable to images of any type of image of a cell orimage of a biological specimen, and is applicable to determinations oftype, density and location for any type of cell or group of cells.Moreover, besides medical applications such as anatomical or clinicalpathology, prostrate/lung cancer diagnosis, etc., the same methods maybe performed to analysis other types of samples such as remote sensingof geologic or astronomical data, etc. The operations disclosed hereinmay be ported into a hardware graphics processing unit (GPU), enabling amulti-threaded parallel implementation.

FIG. 23 shows a biopsy tissue sample 10 that has been obtained from atissue region of a patient. The tissue sample 10 is sliced intoneighboring tissue slices, such as tissue slices 1, 2, 3 and 4 asillustrated in FIG. 23. The tissue slices may have a thickness in themicrometer range, such as between 1 μm-10 μm, for example 6 μm.

The tissue slices are stained with a single stain, a stain and acounter-stain or multiple stains. This way e.g. the image 231 (cf. FIG.2) that is stained by a stain and a counter-stain is obtained as well asa multi-channel image 5.

The multi-channel image 5 may be obtained from one of the tissue slices1, 2, 3 and 4 that is stained by multiple stains, e.g. multiplex slide121 of FIG. 1B that may carry one of the tissue slices. In additionfurther images may be acquired from the stained tissue slices such assingle stain images 6 and 7. These images 231, 5, 6 and 7 may be storedin the electronic memory of an image processing system, such as in theelectronic memory of a computer 101 (cf. FIG. 1A), which may be a servercomputer.

An automatic field of view definition may be performed with respect toone or more of the multiple images, such as with respect to the image231 which results in the thresholded image 237 in which the fields ofview 240.1 and 240.2 are indicated by respective rectangular boxes inaccordance with the embodiment of FIG. 2. The image 5 is unmixed whichprovides a set of unmixed images 5.1, 5.2 and 5.3 assuming, withoutlimitation of generality, that N=3 (cf. FIG. 1B). It is to be noted thatthe unmixed images 5.1, 5.2 and 5.3 share exactly the same coordinatesystem as they are all obtained from the same multi-channel image 5 suchthat no image registration or image alignment is required with respectto the this set of images. The additional images 6 and 7 may or may notundergo an image processing operation.

The images 231/237, 5, 6 and 7 are then registered and aligned using animage registration algorithm. For example, the multi-channel image 5 isselected as a reference image for performing the image registrationalgorithm. The image registration algorithm generates a geometricaltransformation of each one of the other images, i.e. images 231/237, 6and 7 with respect to the multi-channel image 5. Using the multi-channelimage 5 as a reference image for the registration has the advantage thatonly 3 alignment operations need be executed in the example consideredhere. In comparison, when e.g. image 7 would have been selected as thereference image, 5 alignment operations would be required to transformthe images 231/237, 5.1, 5.2, 5.3 and 6 for alignement with image 7.Hence, selecting the multi-channel image 5 as the referencesubstantially reduces the computational burden and reduces latency timesfor the image alignments.

For example, a mapping is generated for each one of the other images231/237, 6 and 7 to the reference image 5 such as a mapping for mappingeach pixel of the image 231/237 to a respective pixel in the image 5, amapping for mapping each pixel of the image 6 to a respective pixel inthe multi-channel image 5, etc. In the example considered here thisresults in three mappings. It is to be noted that the mapping formapping image 231/237 to the multi-channel image 5 can be obtained usingeither image 231 or image 237 as these two images share the samecoordinate system due to the unmixing step performed in accordance withFIG. 2.

The geometrical transformations, i.e. the mappings in the exampleconsidered here, that are obtained as a result of the image registrationare then utilized to align the images 237, 6 and 7 with respect to thereference image, i.e. the multi-channel image 5/unmixed images 5.1, 5.2and 5.3.

These aligned images are displayed on display 8 such as of computer 101(cf. the embodiment of FIG. 1) or the display of a mobilebattery-powered telecommunication device, such as a smartphone, runningan Android or iOS operating system, for example. In the latter case theimages 237, 5.1, 5.2, 5.3, 6, 7 and the geometrical transformations,e.g. the mappings, obtained from the image registration and meta databeing indicative of the fields of view 240.1 and 240.2 are transmittedvia a telecommunication network, such as a mobile cellular digitaltelecommunication network e.g. in accordance with the GSM, UMTS, CDMA orLong-Term Evolution standard, to the mobile battery-poweredtelecommunication device. The display 8 may be touch-sensitive whichenables to enter commands via the graphical user interface of thecomputer 101 or telecommunication device by means of gesturerecognition.

In one embodiment the user may select one of the fields of view bytouching the respective geometrical object, i.e. a rectangular box, thatsymbolizes the field of view. As illustrated in FIG. 23 by way ofexample only this may be the field of view 240.1 on which the userplaces one of his or her fingers 14. In response to this gesture, a zoomin image transformation is executed by magnifying the field of view asalso illustrated in FIG. 23.

An identical zoom in transformation is synchronously executed withrespect to the other images 5.1, 5.2, 5.3, 6 and 7: The field of view240.1 corresponds to image portions 9, 10, 11, 12, 13 in the images 5.1,5.2, 5.3, 6 and 7, respectively. These image portions 9 to 13 are givingby the respective geometrical transformations obtained from the imageregistration, i.e. the mappings. In response to the user's gesture, i.e.touching field of view 240.1 with finger 14, the zoom in imagetransformation that is executed with respect to the field of view 240.1is synchronously also executed with respect to the image portions 9 to13.

FIG. 24 shows an alternative embodiment where a pinch gesture isutilized to zoom in or zoom out. The user may select a portion of one ofthe images, such as of image 5.1 by placing two fingers 14 and 15 on thedisplay 8 thus defining a rectangular region 16. This rectangular region16 corresponds to co-located image regions 17 to 21 in the other images237, 5.2, 5.3, 6 and 7, respectively, which are given by the geometricaltransformations obtained from the image registration, i.e. the mappings.Regions 18 and 19 are identical to region 16 as images 5.1, 5.2 and 5.3share the identical coordinate system.

By distancing the fingers 15 and 14 as illustrated in FIG. 24 a zoom inis executed with respect to region 16 which provides magnified imageportion 16′ and synchronously also with respect to the other co-locatedregions 17-21, which provides the magnified regions 17′, 18′, 19′, 20′and 21′. A zoom out can be performed analogously by reducing thedistance of fingers 14 and 15.

Computers typically include known components, such as a processor, anoperating system, system memory, memory storage devices, input-outputcontrollers, input-output devices, and display devices. It will also beunderstood by those of ordinary skill in the relevant art that there aremany possible configurations and components of a computer and may alsoinclude cache memory, a data backup unit, and many other devices.Examples of input devices include a keyboard, a cursor control devices(e.g., a mouse), a microphone, a scanner, and so forth. Examples ofoutput devices include a display device (e.g., a monitor or projector),speakers, a printer, a network card, and so forth. Display devices mayinclude display devices that provide visual information, thisinformation typically may be logically and/or physically organized as anarray of pixels. An interface controller may also be included that maycomprise any of a variety of known or future software programs forproviding input and output interfaces. For example, interfaces mayinclude what are generally referred to as “Graphical User Interfaces”(often referred to as GUI's) that provide one or more graphicalrepresentations to a user. Interfaces are typically enabled to acceptuser inputs using means of selection or input known to those of ordinaryskill in the related art. The interface may also be a touch screendevice. In the same or alternative embodiments, applications on acomputer may employ an interface that includes what are referred to as“command line interfaces” (often referred to as CLI's). CLI's typicallyprovide a text based interaction between an application and a user.Typically, command line interfaces present output and receive input aslines of text through display devices. For example, some implementationsmay include what are referred to as a “shell” such as Unix Shells knownto those of ordinary skill in the related art, or Microsoft WindowsPowershell that employs object-oriented type programming architecturessuch as the Microsoft .NET framework.

Those of ordinary skill in the related art will appreciate thatinterfaces may include one or more GUI's, CLI's or a combinationthereof. A processor may include a commercially available processor suchas a Celeron, Core, or Pentium processor made by Intel Corporation, aSPARC processor made by Sun Microsystems, an Athlon, Sempron, Phenom, orOpteron processor made by AMD Corporation, or it may be one of otherprocessors that are or will become available. Some embodiments of aprocessor may include what is referred to as multi-core processor and/orbe enabled to employ parallel processing technology in a single ormulti-core configuration. For example, a multi-core architecturetypically comprises two or more processor “execution cores”. In thepresent example, each execution core may perform as an independentprocessor that enables parallel execution of multiple threads. Inaddition, those of ordinary skill in the related will appreciate that aprocessor may be configured in what is generally referred to as 32 or 64bit architectures, or other architectural configurations now known orthat may be developed in the future.

A processor typically executes an operating system, which may be, forexample, a Windows type operating system from the Microsoft Corporation;the Mac OS X operating system from Apple Computer Corp.; a Unix orLinux-type operating system available from many vendors or what isreferred to as an open source; another or a future operating system; orsome combination thereof. An operating system interfaces with firmwareand hardware in a well-known manner, and facilitates the processor incoordinating and executing the functions of various computer programsthat may be written in a variety of programming languages. An operatingsystem, typically in cooperation with a processor, coordinates andexecutes functions of the other components of a computer. An operatingsystem also provides scheduling, input-output control, file and datamanagement, memory management, and communication control and relatedservices, all in accordance with known techniques.

System memory may include any of a variety of known or future memorystorage devices that can be used to store the desired information andthat can be accessed by a computer. Computer readable storage media mayinclude volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer readable instructions, data structures, program modules, orother data. Examples include any commonly available random access memory(RAM), read-only memory (ROM), electronically erasable programmableread-only memory (EEPROM), digital versatile disks (DVD), magneticmedium, such as a resident hard disk or tape, an optical medium such asa read and write compact disc, or other memory storage device. Memorystorage devices may include any of a variety of known or future devices,including a compact disk drive, a tape drive, a removable hard diskdrive, USB or flash drive, or a diskette drive. Such types of memorystorage devices typically read from, and/or write to, a program storagemedium such as, respectively, a compact disk, magnetic tape, removablehard disk, USB or flash drive, or floppy diskette. Any of these programstorage media, or others now in use or that may later be developed, maybe considered a computer program product. As will be appreciated, theseprogram storage media typically store a computer software program and/ordata. Computer software programs, also called computer control logic,typically are stored in system memory and/or the program storage deviceused in conjunction with memory storage device. In some embodiments, acomputer program product is described comprising a computer usablemedium having control logic (computer software program, includingprogram code) stored therein. The control logic, when executed by aprocessor, causes the processor to perform functions described herein.In other embodiments, some functions are implemented primarily inhardware using, for example, a hardware state machine. Implementation ofthe hardware state machine so as to perform the functions describedherein will be apparent to those skilled in the relevant arts.Input-output controllers could include any of a variety of known devicesfor accepting and processing information from a user, whether a human ora machine, whether local or remote. Such devices include, for example,modem cards, wireless cards, network interface cards, sound cards, orother types of controllers for any of a variety of known input devices.Output controllers could include controllers for any of a variety ofknown display devices for presenting information to a user, whether ahuman or a machine, whether local or remote. In the presently describedembodiment, the functional elements of a computer communicate with eachother via a system bus. Some embodiments of a computer may communicatewith some functional elements using network or other types of remotecommunications. As will be evident to those skilled in the relevant art,an instrument control and/or a data processing application, ifimplemented in software, may be loaded into and executed from systemmemory and/or a memory storage device. All or portions of the instrumentcontrol and/or data processing applications may also reside in aread-only memory or similar device of the memory storage device, suchdevices not requiring that the instrument control and/or data processingapplications first be loaded through input-output controllers. It willbe understood by those skilled in the relevant art that the instrumentcontrol and/or data processing applications, or portions of it, may beloaded by a processor, in a known manner into system memory, or cachememory, or both, as advantageous for execution. Also, a computer mayinclude one or more library files, experiment data files, and aninternet client stored in system memory. For example, experiment datacould include data related to one or more experiments or assays, such asdetected signal values, or other values associated with one or moresequencing by synthesis (SBS) experiments or processes. Additionally, aninternet client may include an application enabled to access a remoteservice on another computer using a network and may for instancecomprise what are generally referred to as “Web Browsers”. In thepresent example, some commonly employed web browsers include MicrosoftInternet Explorer available from Microsoft Corporation, Mozilla Firefoxfrom the Mozilla Corporation, Safari from Apple Computer Corp., GoogleChrome from the Google Corporation, or other type of web browsercurrently known in the art or to be developed in the future. Also, inthe same or other embodiments an Internet client may include, or couldbe an element of, specialized software applications enabled to accessremote information via a network such as a data processing applicationfor biological applications.

A network may include one or more of the many various types of networkswell known to those of ordinary skill in the art. For example, a networkmay include a local or wide area network that may employ what iscommonly referred to as a TCP/IP protocol suite to communicate. Anetwork may include a network comprising a worldwide system ofinterconnected computer networks that is commonly referred to as theInternet, or could also include various intranet architectures. Those ofordinary skill in the related arts will also appreciate that some usersin networked environments may prefer to employ what are generallyreferred to as “firewalls” (also sometimes referred to as PacketFilters, or Border Protection Devices) to control information traffic toand from hardware and/or software systems. For example, firewalls maycomprise hardware or software elements or some combination thereof andare typically designed to enforce security policies put in place byusers, such as for instance network administrators, etc.

The foregoing disclosure of the exemplary embodiments of the presentsubject disclosure has been presented for purposes of illustration anddescription. It is not intended to be exhaustive or to limit the subjectdisclosure to the precise forms disclosed. Many variations andmodifications of the embodiments described herein will be apparent toone of ordinary skill in the art in light of the above disclosure. Thescope of the subject disclosure is to be defined only by the claimsappended hereto, and by their equivalents.

Further, in describing representative embodiments of the present subjectdisclosure, the specification may have presented the method and/orprocess of the present subject disclosure as a particular sequence ofsteps. However, to the extent that the method or process does not relyon the particular order of steps set forth herein, the method or processshould not be limited to the particular sequence of steps described. Asone of ordinary skill in the art would appreciate, other sequences ofsteps may be possible. Therefore, the particular order of the steps setforth in the specification should not be construed as limitations on theclaims. In addition, the claims directed to the method and/or process ofthe present subject disclosure should not be limited to the performanceof their steps in the order written, and one skilled in the art canreadily appreciate that the sequences may be varied and still remainwithin the spirit and scope of the present subject disclosure.

What is claimed is:
 1. An image processing method for displaying multiple images of a tissue region of a biopsy tissue sample comprising: acquiring multiple images from tissue slices of the tissue region; performing an image registration algorithm with respect to the multiple images; aligning the images in accordance with the image registration; concurrently displaying the aligned images on a display in a two-dimensional plane; receiving an image transformation command via a graphical user interface with respect to one of the displayed images; and simultaneously executing the image transformation command for all of the displayed images.
 2. The image processing method of claim 1, the tissue slices being marked by stains for the identification of biological features, one of the tissue slices being marked by multiple stains for the acquisition of a multi-channel image, the method further comprising unmixing the multi-channel image to provide a set of unmixed images, wherein the image registration is performed by using the multi-channel image as a reference image for the image registration algorithm to provide a mapping of each one of the multiple images to the reference image, wherein the aligned images and the unmixed images are concurrently displayed in the two-dimensional plane, and wherein the image transformation is simultaneously executed using the respective mappings.
 3. The method of claim 1, wherein the display is touch sensitive and the graphical user interface being configured to perform a gesture recognition of a user's gesture for entry of the image transformation command.
 4. The method of claim 3, wherein the image transformation is zoom in or zoom out and the user's gesture is a pinch gesture that is performed by placing two fingers onto one of the displayed images such that the image transformation is synchronously executed for the image on which the two fingers are placed and all other displayed images.
 5. The method of claim 1, wherein the acquired multiple images are stored on a server computer and further comprising transmitting the acquired images from the server computer to a mobile battery-powered telecommunication device via a telecommunication network for displaying the images on a display of the telecommunication device.
 6. The method of claim 5, wherein image registration is performed by the server computer, and further comprising transmitting the mappings of each one of the multiple images to the reference image from the server computer to the telecommunication device via the network.
 7. The method of claim 2, wherein one of the images is stained by means of a stain and its respective counter-stain, wherein the one of the images is unmixed which provides unmixed images further comprising: spatial low pass filtering of at least one of the unmixed images, local maximum filtering of the at least one of the spatial low pass filtered unmixed images, thresholding the at least one of the spatial low pass filtered unmixed images to identify at least one set of neighboring pixels, defining a field of view by extracting an image portion of the multi-channel image from an image location given by the set of neighboring pixels, the field of view having a predetermined size and shape, displaying a graphical symbol in the at least one of the unmixed images, the graphical symbol being representative of the field of view, wherein the image transformation command is received with respect to the field of view by selecting the graphical symbol via the user interface.
 8. The method of claim 7, further comprising: segmentation of another one of the unmixed images for identification of tissue regions to provide a tissue region mask, and masking the multi-channel image or the at least one of the unmixed images with the tissue mask.
 9. The method of claim 7, wherein a zooming transformation of the field of view is performed in response to the selection of the graphical symbol synchronously in all of the displayed images, wherein the zooming transformation causes an enlargement of the field of view in the at least one of the unmixed images and of respective image portions of the other images that are aligned with the field of view due to the image registration.
 10. The method of claim 9, wherein the selection of the graphical symbol is performed by placing a finger on the graphical symbol.
 11. A system for simultaneously displaying multiple views of a same region of a biological tissue sample, the system comprising: a. a processor; and b. a memory coupled to the processor, the memory stores computer-readable instructions that, when executed by the processor, cause the system to perform operations comprising: i. receiving a plurality of preprocessed images of the biological tissue sample, wherein each preprocessed image shows a view mode of a same region of the biological tissue sample, and wherein each preprocessed image has metadata that describes a preprocessed image local reference frame (PI-LRF) with respect to a global standard reference frame (GSRF); ii. choosing a common display reference frame (CDRF) that is used for image visualization; iii. converting the plurality of preprocessed images to the CDRF by constructing a destination view for each preprocessed image of the plurality of preprocessed images to produce a plurality of displayable images; iv. arranging the plurality of displayable images into a display pattern for viewing on a display screen; v. displaying the plurality of displayable images in the display pattern on the display screen; and vi. accepting user gestures to dynamically alter the CDRF; wherein displaying of the plurality of displayable images allows for simultaneous dynamic viewing of different aspects of the imaged biological tissue sample, and wherein re-converting the plurality of preprocessed images after each user gesture causes all displayable images to simultaneously perform apparent coordinated translation, rotation, or magnification changes.
 12. The system of claim 11, wherein the PI-LRF is a two-dimensional reference frame used to describe a location of a pixel in the preprocessed image.
 13. The system of claim 12, wherein the GSRF is an agreed-upon fixed two-dimensional reference frame used to describe a space of pixel locations and which allows an understanding of spatial relationships between different images by defining affine mappings between each image local reference frame (I-LRF) and the GSRF.
 14. The system of claim 13, wherein the metadata of each preprocessed image describe the spatial location, orientation, and magnification of the preprocessed image with respect to the GSRF, wherein the metadata defines a first affine mapping between the I-LRF and the GSRF.
 15. The system of claim 14, wherein the operation of choosing the CDRF further comprises: a. creating a two-dimensional display image pixel grid, wherein grid intersections denote pixel locations; b. constructing a two-dimensional display image local reference frame (DI-LRF) used to describe pixel locations in the display image pixel grid; c. choosing a location, orientation, and magnification for the DI-LRF with respect to the GSRF; and d. computing an affine transform that maps pixel locations in the DI-LRF to locations in the GSRF; whereby this construction serves as a display image template and provides an affine partial mapping for production of display images.
 16. The system of claim 15, wherein the operation of converting the plurality of preprocessed images to the CDRF further comprises: a. constructing a working copy of the CDRF display image template and affine partial mapping, wherein the working copy of the display image template comprises memory cells to hold pixel values for a display image; b. composing the affine partial mapping with the first affine mapping for the preprocessed image to produce a composite mapping that transforms pixel locations in the DI-LRF of the display image to a location in the PI-LRF of the preprocessed image; and c. painting the display image by performing operations for each display image pixel comprising: i. mapping with the composite affine transform from a DI-LRF location of the display image pixel to a location in the PI-LRF of the preprocessed image; ii. interpolating a pixel value among neighboring pixels in the preprocessed image around that mapped location; and iii. delivering the interpolated pixel value as the pixel value used in the display image at the display image pixel; wherein performing operations for each display image pixel transforms each preprocessed image to a display image for representation on the display screen.
 17. The system of claim 12, wherein the two-dimensional local reference frames and the agreed-upon fixed two-dimensional reference frame are orthogonal Cartesian reference frames.
 18. The system of claim 11, wherein the operation of converting the plurality of preprocessed images further performs nonlinear corrections on the plurality of preprocessed images to remove optical distortion.
 19. The system of claim 11, wherein a preprocessing of images of the biological tissue sample produces the plurality of preprocessed images that each contains metadata.
 20. The system of claim 11, wherein the operations further comprise translating the plurality of images in unison on the display screen in response to an input gesture from an interface device, wherein translating the plurality of images provides a desired perspective of the imaged biological tissue sample.
 21. The system of claim 11, wherein the operations further comprise rotating the plurality of images in unison on the display screen in response to an input gesture from an interface device, wherein rotating the plurality of images provides a desired perspective of the imaged biological tissue sample.
 22. The system of claim 11, wherein the operations further comprise zooming in and out of the plurality of images in unison on the display screen in response to an input gesture from an interface device, wherein zooming of the plurality of images provides a desired perspective of the imaged biological tissue sample.
 23. The system of claim 11, wherein the operations further comprise removing one or more images from the plurality of images on the display screen, wherein removing the one or more images declutters the display screen.
 24. The system of claim 11, wherein the operations further comprise adding new mode images onto the display screen, wherein the new mode images is viewed in tandem with other image modes.
 25. The system of claim 11, wherein the operations further comprise rearranging the display pattern to form an alternative display pattern, wherein the alternative display pattern brings together image modes for closer inspection.
 26. The system of claim 11, wherein the operations further comprise stacking two or more image modes in response to an input gesture from an interface device, wherein the two or more image modes are translucent, wherein stacking the two or more image modes reinforces image features.
 27. The system of claim 11, wherein the operations further comprise saving the display pattern of a current examination as a saved template, wherein the saved template facilitates displaying of a plurality of images in future examinations.
 28. A non-transitory computer-readable medium storing instructions which, when executed by one or more processors of a system for simultaneously displaying multiple views of a same region of a biological tissue sample, cause the system to perform a method comprising: receiving a plurality of preprocessed images of the biological tissue sample, wherein each preprocessed image shows a view mode of a same region of the biological tissue sample, and wherein each preprocessed image has metadata that describes a preprocessed image local reference frame (PI-LRF) with respect to a global standard reference frame (GSRF); choosing a common display reference frame (CDRF) that is used for image visualization; converting the plurality of preprocessed images to the CDRF by constructing a destination view for each preprocessed image of the plurality of preprocessed images to produce a plurality of displayable images; arranging the plurality of displayable images into a display pattern for viewing on a display screen; displaying the plurality of displayable images in the display pattern on the display screen; and accepting user gestures to dynamically alter the CDRF; wherein displaying of the plurality of displayable images allows for simultaneous dynamic viewing of different aspects of the imaged biological tissue sample, and wherein re-converting the plurality of preprocessed images after each user gesture causes all displayable images to simultaneously perform apparent coordinated translation, rotation, or magnification changes. 